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Abstract 


Mobile  learning  and  embedded  training  technologies  are  being  fielded  by  military  organizations 
to  ensure  that  learners  can  access  training  material  anytime  and  anywhere.  Recent  advances  in 
learning  technologies  and  future  weapon  concepts  offer  the  potential  to  offer  anytime,  anywhere 
training  to  new  skill  domains.  The  potential  of  these  technologies  to  improve  small  arms  coaching 
skills  was  investigated  to  test  the  willingness  of  soldiers  to  utilize  anytime,  anywhere  self-directed 
training  and  to  determine  the  utility  and  usability  of  embedding  small  arms  coaching  training  in 
future  weapons. 

In  this  experiment  soldiers  were  first  tested  on  their  ability  to  identify  shooting  errors  depicted  in 
videos  before  they  were  issued  with  either  a  paper  or  digital  small  arms  coaching  training  aid.  The 
soldiers  then  retained  the  training  aid  for  five  days  and  were  instructed  to  review  the  training 
material  to  prepare  for  a  final  test.  At  the  final  test  session  soldier  reported  they  seldom  used  the 
training  aid,  yet  their  error  detection  ability  improved.  The  gain  was  attributed  to  the  testing  effect 
of  watching  the  videos,  and  interpreted  as  providing  some  evidence  for  the  efficacy  of  digital 
media  for  training  small  arms  coaching  skills.  The  limited  use  of  the  training  aids  during  the 
intervening  period  indicates  that  soldiers  will  not  necessarily  conduct  anytime,  anywhere 
self-directed  training. 

Usability  questionnaires  and  focus  group  discussions  indicate  that  the  soldiers  prefer  a  combination 
of  paper  and  digital  small  arms  coaching  training  aids.  It  is  recommended  that  a  prototype  small 
arms  coaching  aid  be  developed  and  that  social  support  be  provided  where  self-directed  learning  is 
expected  of  soldiers. 


Significance  to  defence  and  security 


Small  arms  coaching  is  a  skill  that  all  members  of  the  Canadian  Armed  Forces  must  demonstrate 
in  the  Annual  Personal  Weapons  Test.  Increasing  the  supply  of  skilled  small  arms  coaches  would 
obtain  better  value  from  existing  training  resources  and  provide  the  potential  for  reduced  training 
costs  or  higher  marksmanship  performance. 

The  multi-media  content  for  small  arms  coaching  training  was  demonstrated  to  significantly 
improve  the  ability  to  diagnose  shooting  errors.  Soldiers  will  not  necessarily  use  mobile  learning 
technologies  to  conduct  self-directed  training  on  their  own  time,  however.  If  the  Canadian  Armed 
Forces  makes  greater  use  of  self-directed  learning  and  mobile  learning,  the  technology  should  be 
accompanied  by  social  supports  that  motive  soldiers  to  use  the  technology. 
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Resume 


Les  technologies  d’apprentissage  electronique  sans  fil  et  d’instruction  integree  sont  deployees  sur 
le  terrain  par  les  organisations  militaires  afm  de  s’ assurer  que  les  apprenants  puissent  avoir  acces 
au  materiel  d’instruction  en  tout  temps  et  n’importe  oil.  Les  progres  recents  dans  les  technologies 
d’apprentissage  et  les  concepts  d’armes  futures  permettent  d’offrir  n’importe  ou  n’importe  quand 
une  formation  dans  de  nouveaux  domaines  de  competence.  Nous  avons  etudie  le  potentiel  qu’ont 
ces  technologies  d’ameliorer  les  competences  pedagogiques  de  tir  a  l’arme  legere  dans  le  but  de 
tester  l’empressement  des  soldats  a  pratiquer  1’ autoformation  n’importe  oil,  n’importe  quand  et  de 
determiner  l’utilite  et  l’utilisabilite  d’integrer  la  formation  d’instructeur  de  tir  a  l’arme  legere, 
pour  les  futures  armes. 

Lors  de  cette  experience,  les  soldats  ont  ete  evalues  quant  a  leur  capacite  a  cerner  les  erreurs  de  tir 
dans  des  videos  avant  de  recevoir  le  materiel  de  formation  d’instructeur  de  tir  a  l’arme  legere,  en 
copie  papier  ou  en  version  numerique.  Les  soldats  ont  conserve  ce  materiel  pendant  cinq  jours  et 
avaient  re?u  la  directive  de  le  passer  en  revue  pour  se  preparer  a  1’ epreuve  finale.  Lors  de  cette 
epreuve,  les  soldats  ont  indique  qu’ils  avaient  rarement  utilise  le  materiel,  mais  leur  capacite  a 
cerner  les  erreurs  s’etait  amelioree.  Cette  amelioration  etait  attribuable  a  l’effet  du  test  de 
visionnement  des  videos  et  consideree  comme  une  preuve  de  l’efficacite  des  medias  numeriques 
sur  les  competences  pedagogiques  au  tir  d’armes  legeres.  L’usage  limite  du  materiel  d’instruction 
lors  de  cette  periode  intermediaire  indique  que  les  soldats  ne  suivront  pas  necessairement  une 
autoformation  n’importe  oil  n’importe  quand. 

Les  questionnaires  et  les  discussions  des  groupes  temoins  sur  l’utilisabilite  indiquent  que  les 
soldats  preferent  une  combinaison  de  materiel  d’instruction  sur  papier  et  numerique  pour  la 
formation  d’instructeur  de  tir  a  l’arme  legere.  II  est  recommande  d’elaborer  un  prototype  de 
materiel  pedagogique  pour  les  instructeurs  de  tir  a  l’arme  legere  et  d’offrir  un  soutien  social 
lorsque  les  soldats  sont  censes  suivre  une  autoformation. 


Importance  pour  la  defense  et  la  securite 


L’instructeur  de  tir  a  l’arme  legere  possede  une  competence  que  tous  les  membres  des  Forces 
armees  canadiennes  doivent  demontrer  lors  de  1’ epreuve  annuelle  de  tir  a  l’arme  personnelle. 
Accroitre  le  nombre  d’ instructeurs  de  tir  a  l’arme  legere  experiments  permettrait  d’optimiser  les 
outils  pedagogiques  existants  et  de  reduire  potentiellement  les  couts  de  formation  ou  d’obtenir  de 
meilleures  performances  de  tir. 

Le  contenu  multimedia  de  la  formation  d’instructeur  de  tir  a  l’arme  legere  a  demontre  qu’il 
ameliorait  considerablement  la  capacite  de  cerner  les  erreurs  de  tir.  Toutefois,  les  soldats 
n’utiliseront  pas  necessairement  les  technologies  d’apprentissage  sans  fil  pour  s’assurer  une 
autoformation  durant  leurs  temps  libres.  Si  les  Forces  armees  canadiennes  pratiquent  davantage 
l’autoformation  et  l’apprentissage  sans  fil,  la  technologie  devrait  etre  accompagnee  de  mesures  de 
soutien  social  qui  encouragent  les  soldats  a  utiliser  la  technologie. 
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1  Introduction 


The  Canadian  Army  anticipates  acquiring  a  new  generation  of  carbines,  rifles,  and  machine  guns. 
This  new  generation  of  small  arms  could  include  technologies  that  have  not  been  previously 
applied  to  small  arms,  such  as  computerized  fire  control  and  target  sharing.  These  new 
technologies  are  expected  to  substantially  increase  the  effectiveness  of  the  weapon  and  soldier. 
To  assist  with  this  acquisition,  Defence  Research  and  Development  Canada’s  project  on  Future 
Small  Arms  is  examining  the  implications  of  these  emerging  technologies  for  Canadian  Army 
operations  (Flarris,  Levesque,  Andrukaitis,  &  Laou,  2013;  Tack  &  Angel,  2014). 

If  new  small  arms  technologies  are  provided  for  use  by  soldiers,  they  will  likely  require  additional 
skills  to  employ  them  (Papaz,  Angel,  &  Thomson,  2013).  Flowever,  soldiers  and  units  frequently 
report  that  they  have  difficulty  conducting  the  existing  amount  of  training  (Tack,  Nakaza,  & 
Osborne,  2013).  Scheduling  groups  of  trainees  and  instructors,  drawing  weapons  and  ammunition 
from  stores,  travel  to  training  facilities,  support  personnel,  and  clean-up,  for  example,  contribute 
to  the  cost  and  difficulty  of  conducting  marksmanship  training.  Substituting  a  central  simulation 
facility  for  a  live  fire  range,  can  reduce  some,  but  not  all  of  these  overheads.  If  training  soldiers  to 
use  the  new  technologies  demands  greatly  increased  training  resources,  this  demand  could 
compromise  the  introduction  of  future  small  arms  into  the  Canadian  Armed  Forces  (CAF).  Ideally 
the  new  technologies  should  make  training  faster  and  easier. 

Defence  Research  and  Development  Canada  (DRDC)’s  Future  Small  Arms  Research  (FSAR) 
project  is  investigating  the  potential  of  utilizing  the  computing,  sensing,  and  display  capabilities 
expected  in  the  future  weapon  to  provide  embedded  training  (ET)  in  the  new  weapon  (Angel  & 
Tack,  2013;  Bruyn-Martin  &  Brown,  2013).  ET  provides  the  opportunity  to  train  in  the  absence 
of  centralized,  dedicated  training  facilities  and  equipment.  In  this  way  training  can  take  place 
wherever  and  whenever  the  training  audience  and  circumstances  allow  (Bruyn-Martin  &  Brown, 
2013;  Finley,  Alderman,  Peckham,  &  Strasel,  1988;  NATO,  2009). 

The  ability  of  ET  to  provide  computer-based  training  independent  of  location  is  similar  to  training 
applications  (“apps”)  provided  on  mobile  computing  devices.  Like  ET,  apps  have  the  potential  to 
enable  learning  anytime,  anyplace  and  do  not  entail  the  same  overhead  as  more  centralized, 
synchronous  training  (BenMoussa,  2003;  Virvou  &  Alepis,  2005).  The  ability  to  train  using  the 
time  available  between  other  scheduled  events  (informally  termed  “hurry  up  and  wait”  in  military 
parlance)  is  one  reason  why  mobile  learning  is  seen  as  beneficial  for  military  training  audiences 
(Department  of  the  Army,  2011;  Flaag,  2011).  Guidance  for  military  use  of  mobile  learning,  and 
particularly  in  infantry  contexts,  is  less  common  (Tucker,  2010),  however.  It  is  known  that  the 
context  of  mobile  device  use  is  critical  to  its  usability  (Biel,  20 1 0),  but  the  available  guidance  for 
mobile  learning  is  typically  derived  from  experiences  in  academic  or  commercial  contexts  (Ally, 
2009;  Kukulska-Hulme,  2007;  Motiwalla,  2007).  The  US  Army  Research  Institute  Keller-Glaze, 
H.,  Horey,  J.,  Nicely,  K.,  Brusso,  R.,  Miller  Nihill,  M.,  &  Cobb,  M.  G.,  2013)  and  a  community 
of  practice  (Advanced  Distributed  Learning  Initiative,  2013)  have  produced  advice  for  the 
development  of  mobile  learning  for  military  populations,  but  the  advice  is  not  aimed  at  the 
infantry  soldier  in  the  field.  The  infantry  context,  such  as  limited  or  no  connectivity,  greatly 
varying  environmental  lighting,  exposure  to  weather  and  dirt,  vibration,  rough  handling,  and 
fatigued  users  have  not  been  systematically  assessed  as  influences  on  the  design  of  a  mobile 
application  (Tucker,  2010).  Research  on  ET  for  dismounted  soldiers  has  aimed  at  fielding 


DRDC-RDDC-201 5-R003 


1 


ambitious  integrated  soldier  systems  and  not  at  marksmanship  specifically  (Dumanoir,  Garrity, 
Lowe  &  Witmer,  2002;  Dyer,  2009;  Knerr,  Garrity  &  Lampton,  2004). 

To  inform  the  CAF  on  the  suitability  of  ET  apps  as  a  way  to  train  the  use  of  future  small  arms,  an 
experiment  was  performed  to  assess  the  utility,  or  effectiveness,  of  a  digital  marksmanship 
coaching  application,  either  as  a  learning  tool  or  as  a  job  aid.  Small  arms  coaching  was  selected 
because  it  may  be  needed  more  widely  if  small  arms  training  is  to  be  conducted  away  from 
specialized  marksmanship  training  facilities  and  instructors.  Moreover,  marksmanship  coaching 
skill  does  not  depend  on  training  the  motor  skills  of  marksmanship  (Proctor  &  Woodman,  2007), 
but  rather  the  cognitive  aspects  (Chung,  Nagashima,  Espinosa,  Berka,  &  Baker,  2009a,  2009b) 
and  so  could  be  embedded  into  both  the  weapon  or  any  other  computing  device  issued  to  soldiers. 
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2  Method 


In  this  experiment  soldiers  used  an  app,  a  booklet,  or  only  their  background  knowledge,  to  identify 
shooting  errors  committed  by  a  soldier  depicted  in  video  clips.  It  was  hypothesized  that  soldiers  will 
be  more  accurate  in  identifying  shooting  errors  using  the  app  and  the  booklet  than  with  only  their 
background  knowledge.  To  assess  the  likelihood  that  soldiers  will  use  anytime  -  anywhere  training, 
the  usage  of  the  devices  was  tracked  over  the  course  of  a  week.  It  was  hypothesized  that  having 
constant  access  to  the  learning  content,  along  with  an  imminent  opportunity  to  use  the  knowledge, 
would  result  in  frequent  use  of  the  paper  and  electronic  versions  of  the  material.  Finally,  the 
experiment  used  questionnaires  and  a  focus  group  to  investigate  the  usability  of  marksmanship 
apps,  with  attention  to  the  infantry  context. 

2.1  Participants 

Forty  six  regular  force  male  infantry  and  combat  engineer  soldiers  participated  in  the  study  during 
the  experimental  campaign  held  at  DRDC  Toronto  17  January  to  7  March.  Twenty  three  soldiers 
were  from  1  Battalion,  Royal  Canadian  Regiment  (RCR),  three  from  1  Battalion,  Combat 
Engineering  Regiment  (CER),  10  from  2  Battalion  RCR,  and  10  from  2  Battalion  CER.  The 
sample  consisted  of  four  privates,  34  corporals,  four  master  corporals,  and  four  sergeants.  All  had 
normal  or  corrected  to  normal  vision,  as  assessed  by  Snellen  tests  (6/6m  or  better).  All  were 
Personal  Weapons  Test  (PWT)  three  or  four  qualified  (Department  of  National  Defence,  2007). 
Thirty  (65%)  had  operational  experience  in  Afghanistan,  and  21  (46%)  had  actually  engaged  the 
enemy. 

These  participants  were  all  tasked  by  their  chain  of  command  to  report  to  DRDC  Toronto  for  one 
week  for  the  purpose  of  voluntary  participation  in  various  experiments  and  studies  as  described  in 
Protocol  2013-065  Omnibus  Protocol  for  all  Studies  Run  during  the  Winter  2014  Troops  Tasking 
at  DRDC  Toronto.  They  were  informed  that  they  could  withdraw  from  the  experiment  at  any 
time. 


2.2  Apparatus 

An  electronic  version  of  the  Rifle  Marksmanship  Diagnostic  and  Training  Guide  (James  &  Dyer, 
2011)  developed  by  the  Canadian  Defence  Academy  was  delivered  on  Apple  iPad  Mini  devices 
provided  by  the  investigators.  The  printed  version  of  the  guide  was  presented  in  a  spiral-bound 
format  (Figures  1  and  2). 
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Figure  1:  Apple  iPad  Mini. 


Figure  2:  Spiral-bound  booklet. 


To  measure  the  ability  to  detect  marksmanship  errors,  a  list  of  36  shooting  errors  was  compiled 
from  the  Rifle  Marksmanship  Diagnostic  and  Training  Guide  (James  &  Dyer,  2011)  and  used  to 
produce  the  Shooting  Error  Checklist  presented  in  Annex  A.  Corresponding  test  material  was 
produced  in  the  form  of  videos  clips.  The  video  clips  depicted  the  shooting  errors  identified  in  the 
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guide  as  well  as  error-free  shooting.  The  videos  were  recorded  in  a  studio  and  edited  by  a 
professional  videographer  using  1080p  High  Definition  video  and  audio  format.  One  right-handed 
and  one  left-handed  Canadian  Army  small  arms  instructors  agreed  to  demonstrate  the  errors  in 
video  recording.  A  small  arms  expert  from  the  Canadian  Army  coached  the  models  to  ensure  that 
the  desired  error  and  only  the  desired  error  was  demonstrated  in  each  clip.  The  soldiers  used  an 
operational  weapon,  modified  with  a  compressed  gas  system  that  would  cycle  the  action  of  the 
weapon,  produce  a  small  amount  of  noise,  and  produce  recoil  forces  when  the  trigger  was  pulled. 
In  each  clip  the  soldier  fired  five  times,  and  the  duration  of  each  clip  was  approximately 
15  seconds.  A  total  of  131  video  clips  were  produced,  depicting  the  shooting  errors  committed  by 
right-handed  and  left-handed  shooters.  The  errors  were  also  demonstrated  in  both  the  prone  and 
kneeling  position,  with  the  exception  of  errors  that  were  specific  to  a  particular  posture.  The 
videos  were  then  reviewed  by  four  small  arms  instructors  using  the  Shooting  Error  Checklist.  The 
judges  showed  substantially  greater  agreement  than  would  be  expected  by  chance,  as  measured 
by  Fleiss’  Kappa,  Kobserved  =  0.524,  Kexpected  =  0.037.  Only  clips  where  the  same  error  identification 
was  assigned  by  three  or  four  of  the  judges  were  retained  for  use  in  the  experiment. 

The  videos  were  presented  on  Sharp  PNL702B  70-inch  LED  monitors. 

Three  questionnaires  were  used  to  evaluate  the  usability  of  the  both  the  printed  and  digital  versions 
of  the  guide.  The  first  was  Brooke’s  (1996)  System  Usability  Scale,  which  is  a  validated  (Bangor, 
Kortum,  &  Miller,  2008)  instrument  that  provides  a  very  quick  characterization  of  the  usability  of 
any  technological  system.  They  also  completed  a  subset  of  the  Mobile  Phone  Usability 
Questionnaire  (Ryu  &  Smith-Jackson,  2005)  which  is  a  validated  (Ryu  &  Smith- Jackson,  2006) 
instrument  that  asks  detailed  questions  about  mobile  phone  usability.  Questions  that  pertained 
solely  to  mobile  telephones,  e.g.,  ringtones,  were  not  included  in  the  version  presented  to  the 
subjects.  The  final  questionnaire  was  the  Context  and  Feature  questionnaire.  This  questionnaire  is 
a  checklist  and  a  free  form  instrument  developed  for  this  experiment  from  Jumisko-Pyykko’s  and 
Vainio’s  (2011)  model  of  mobile  computing  context.  The  purpose  of  the  questionnaire  was  to 
identify  the  circumstances  where  infantry  soldiers  might  use  paper  or  digital  learning  materials. 
These  questionnaires  are  presented  in  Annex  B. 

2.3  Procedure 

Visual  acuity,  contrast  sensitivity,  and  colour  vision  data  were  collected  under  protocol  2013-073: 
Automated  Target  Cueing  in  Scene  Search  (Dr.  Mackenzie  Glaholt,  DRDC  Toronto  Research 
Centre)  and  shared  with  the  principal  Investigator  of  this  study.  The  demographic  data  were 
collected  under  protocol  2013-071:  Effect  of  Load,  Bulk  and  Stiffness  of  Soldier  Equipment  on 
Physical  Performance:  Canadian  Load  Effects  Assessment  Program  (CAN-LEAP)  (Dr.  Michel 
Duchanne,  DRDC  Toronto  Research  Centre)  and  shared  with  the  principal  Investigator  of  this 
study. 

The  experiment  consisted  of  two  experimental  sessions  which  took  place  in  a  meeting  room  in 
DRDC  Toronto.  The  participants  were  tested  in  groups  of  up  to  10.  One  group  of  participants  was 
tested  each  week  for  six  weeks.  Table  1  provides  an  overview  of  the  experimental  procedure. 
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Table  1:  Overview  of  procedure. 


Session  1 
(Monday) 

4  Day  Interval 

Session  2 
(Friday) 

Introduction  and  informed 

consent 

Participants  retain  guide  and 
review  at  their  convenience 

View  Marksmanship  Videos 
(job  aid  test,  with  guide) 

View  Marksmanship  Videos 
(pre -training  test,  no  guide) 

View  Marksmanship  Videos 
(post  training  test,  no  guide) 

Guide  Familiarization  and 
information  retrieval  task 

Mobile  Phone  Usability 
Questionnaire 

System  Usability  Scale 

Context  Questionnaire 

View  Marksmanship  Videos 
(job  aid  test,  with  guide) 

Focus  Group 

Dismissal  with  instructions  to 
review  guide  when 
convenient 

Debrief  &  Dismiss 

End  Session  1 

End  Session  2 

The  first  session,  conducted  on  the  Monday  morning  of  the  participant’s  week  at  DRDC  Toronto, 
began  with  a  verbal  explanation  of  the  experiment.  The  participants  were  provided  with  an 
information  package  and  consent  form  explaining  participation  in  this  experiment  and  their  right 
to  withdraw  their  consent.  All  the  questions  were  answered  at  this  point.  No  participants 
withdrew  from  the  experiment. 

The  participants  then  viewed  16  video  clips  on  large  screen  display  to  assess  their  pre-existing 
knowledge  of  shooting  errors.  In  some  of  the  clips  the  soldier’s  technique  was  correct  and  in  most 
the  soldier  made  one  or  more  errors  identified  in  the  guide.  Each  clip  (see  Figure  3)  was  presented 
at  least  twice  and  would  be  repeated  again  if  a  subject  asked  to  see  it  again.  The  participants  were 
instructed  to  use  the  Shooting  Error  Checklist  to  indicate  which,  if  any,  shooting  errors  were 
present  in  the  video. 
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Figure  3:  A  frame  from  a  video  clip  depicting  the  rifle  not  resting  at  V  of  thumb  and  palm. 


Following  the  video  clips,  the  participants  were  issued  the  guide,  either  in  a  printed  or  a  digital 
form  (on  an  iPad  Mini),  which  they  retained  for  the  week.  The  purpose  and  features  of  the  guide 
were  explained  by  a  small  arms  expert  from  the  Canadian  Army.  To  familiarize  them  with  the 
guide,  they  were  directed  to  browse  through  various  sections  and  required  to  find  specific 
information.  This  familiarization  specifically  included  the  sections  of  the  guide  that  dealt  with  the 
identification  of  shooting  errors.  To  militate  against  potential  bias  or  instructional  style  changes 
on  the  part  of  the  small  arms  expert,  the  duration  of  the  browsing  activity  was  set  at  1 5  minutes 
and  the  same  search  topics  were  used  for  both  formats.  In  addition,  the  expert  was  cautioned 
against  using  evaluative  statements  when  describing  the  guide.  The  differences  in  the  methods  of 
navigation  through  the  two  formats  prevented  an  absolutely  identical  familiarization  treatment, 
however.  These  precautions  reduce,  but  cannot  guarantee  elimination  of  any  instructional  style  or 
variations  between  formats  on  the  part  of  the  small  arms  expert. 

The  participants  then  viewed  another  set  of  16  video  clips,  this  time  with  the  instruction  that  they 
could  use  the  guide  as  a  job  aid  in  identifying  the  errors.  These  were  all  new  clips,  but  depicting 
the  same  set  of  errors.  The  participants  again  used  the  multiple  choice  sheet  in  Annex  A  to 
indicate  which,  if  any,  errors  they  observed.  The  participants  were  then  told  that  they  should 
retain  the  guide  for  review  at  their  convenience  to  prepare  for  the  second  session  of  the 
experiment.  They  were  then  dismissed. 

The  second  session  of  the  experiment  commenced  on  the  Friday.  The  participants  were  first  asked 
to  report  how  many  times  they  used  the  guide  for  five  or  more  minutes  each  day  during  the 
intervening  period.  They  were  then  instructed  to  again  use  the  guide  as  a  job  aid  while  evaluating 
another  16  video  clips.  At  the  end  of  this  task  they  returned  their  guides  to  the  experimenter.  The 
utility  of  the  guide  as  a  training  aid  was  then  tested  by  asking  the  participants  to  report  shooting 


DRDC-RDDC-201 5-R003 


7 


errors  in  a  final  set  of  16  video  clips.  The  video  clips  used  in  the  Friday  session  were  the  same 
video  clips  used  in  the  Monday  session,  although  they  were  presented  in  a  new  order. 

Following  the  video  clip  portion  of  the  session,  the  soldiers  completed  the  usability 
questionnaires  and  participated  in  a  focus  group  facilitated  by  a  Canadian  military  arms  expert, 
according  to  the  protocol  in  0.  A  research  made  written  notes  during  the  discussion.  At  the 
conclusion  of  the  focus  group,  the  participants  were  debriefed  on  the  experiment  and  dismissed. 
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3  Results 


All  subjects  completed  the  experiment,  with  a  total  of  20  subjects  in  the  groups  receiving  the 
booklet  version  of  the  guide  and  26  subjects  in  the  groups  receiving  the  digital  version.  The 
difference  in  the  number  of  subjects  receiving  each  format  of  the  guide  arose  due  to  the 
availability  of  subjects  and  testing  constraints.  The  differing  sizes  of  the  groups  were  not  under 
the  control  of  the  experimenters  and  laboratory  resource  limitations  required  that  all  subjects  in  a 
group  be  tested  with  the  same  format  of  the  guide. 


3.1  Usage 


The  subjects  reported  very  little  use  of  the  guide  between  the  experiment  sessions.  Subsequently, 
most  subjects  did  not  use  it  at  all.  A  repeated  measures  analysis  of  variance  shows  that  there  was 
not  a  reliable  difference  in  booklet  and  digital  guide  usage,  F(1 44)  =  2.09,  p  >  .05.  There  was  a 
strong  effect  of  repeated  tests,  F(4ii76)  =  24.71,  p  <  .001.  The  interaction  was  not  significant, 
F(4,i76)  =  0.83,  p  >  .05.  A  planned  comparison  was  performed  to  test  for  a  linear  trend.  The 
contrast  was  significant,  F(L44|  =  38.09,  p  >  .001,  indicating  that  usage  declined  during  the  week. 
Inspection  shows  usage  of  both  the  printed  guide  (Figure  4)  and  digital  guide  (Figure  5)  was  most 
frequent  on  Monday,  when  nearly  all  subjects  used  the  guide  at  least  once,  and  most  did  not  refer 
to  it  again  during  the  week. 


25 


20 


=  15 


£  10 


Usage  of  Printed  Guide 


Monday  Tuesday  Wednesday  Thursday 

Number  of  Uses  Per  Day 

BO  ■  1  m2  ■  3  ■  4 


Friday 


Figure  4:  Number  of  soldiers  reporting  how  often  they  used  the  printed  guide  each  day. 
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Usage  of  Digital  Guide 


Monday  Tuesday  Wednesday  Thursday  Friday 

Number  of  Uses  Per  Day 


■  0  ■  1  m2  m  3  4 


Figure  5:  Number  of  soldiers  reporting  how  often  they  used  the  digital  guide  each  day. 

3.2  Utility  as  a  job  aid 

The  experimenters  observed  that  the  subjects  did  not  use  the  guide,  in  either  printed  or  digital 
form,  as  a  job  aid  when  it  was  available  and  when  they  were  encouraged  to  do  so.  Consequently, 
the  data  from  the  testing  sessions  intended  to  measure  the  guide’s  utility  as  a  job  aid  did  not  bear 
upon  the  utility  of  the  guide  as  a  job  aid  and  so  this  analysis  was  not  performed. 

3.3  Utility  as  a  training  aid 

The  proportion  of  correct  video  judgements,  depicted  in  Figure  6,  were  analyzed  using  repeated 
measures  analysis  of  variance  with  guide  media  type  (booklet  versus  digital)  as  a  between 
subjects  factor  and  testing  event  (pretest,  job  aid  test  1,  job  aid  test  2,  post-test)  as  a  within 
subjects  factor.  The  analysis  revealed  that  there  was  no  main  effect  of  guide  type,  F(ii32)  =  0.621, 
p  >  .05  and  no  effect  of  testing  event  F(3i96)  =  1.947,  p  >  .05.  The  overall  interaction  did  not  reach 
significance,  F(3.%)  =  0.377,  p  >  .05.  A  planned  comparison  revealed  that  performance  on  the 
post-test  was  significantly  better  than  performance  on  the  pre-test,  F(1i32)  =  5.07,  p  <  .05,  but  a 
planned  comparison  of  the  pre-post  improvement  of  the  booklet  group  to  the  digital  group 
revealed  no  significant  difference,  F(ij32)  =  0.689,  p  >  .05. 
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3.4  Usability 

The  Mobile  Phone  Usability  Questionnaire  scores  for  each  group  were  computed  by  calculating 
scores  for  each  of  the  sub-scales.  The  scores  for  the  printed  and  digital  guides  were  compared 
using  t-tests.  The  mean  scores  differed  only  on  the  Commands  and  Memory  Load  sub-scale, 
t(42)  =  3.86,  p  <  .001,  where  the  digital  version  of  the  guide  was  seen  as  being  a  better  system 
(mean  rating  of  5.58  vs  4.46  on  a  scale  where  1  =  strongly  disagree  and  7  =  strongly  agree). 

The  System  Usability  Scale  scores  were  compared  using  a  t-test  for  independent  samples.  Both 
versions  of  the  guide  received  moderately  positive  ratings  (mean  score  of  67.84  for  the  printed 
version  and  75.79  for  the  digital  version  on  a  scale  ranging  from  zero  to  100).  A  t-test  for 
independent  groups  found  that  these  ratings  were  not  significantly  different,  t(42)  =  1.68,  p  >.05. 

3.5  Focus  group 

Six  focus  groups  were  conducted.  Following  from  Sim’s  (1998)  observation  that  focus  group 
results  are  indicative  of  the  collective  discussion,  rather  than  multiple  individuals,  the  data  from 
each  focus  group  were  treated  as  a  single  observation.  This  seems  especially  suitable  because  the 
groups  consisted  of  formed  units  with  an  intact  rank  structure  and  differential  levels  of  experience 
with  small  arms  coaching.  The  notes  from  the  discussions  were  then  reviewed  to  derive  categories 
of  responses  to  the  focus  group  protocol  questions.  The  frequencies  of  each  response  type  were 
then  counted  across  each  focus  group  to  provide  a  simple  content  frequency  analysis 
(Onwuegbuzie,  Dickinson,  Leech,  &  Zoran,  2009).  Annex  C  presents  the  categorized  responses  to 
the  discussion  topics  along  with  the  frequencies  of  each  response.  The  maximum  possible 
frequency  is  6,  which  indicates  that  the  response  was  provided  by  all  focus  groups. 
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All  groups  believe  there  was  a  need  for  a  small  arms  coaching  training  aid.  Three  of  the  six 
groups  also  felt  there  was  not  a  need  because  the  material  was  available  elsewhere.  The  groups 
were  also  unanimous  in  their  desire  for  a  single  version  of  the  training  aid  that  would  be 
subdivided  into  sections  for  novices  and  advanced  users.  They  believed  that  all  different  types  of 
users  (e.g.,  Navy,  reservists,  instructors)  would  be  served  by  either  the  novice  or  advanced 
sections  of  a  unified  training  aid.  When  asked  whether  a  training  aid  should  be  in  paper  or  digital 
format,  four  groups  desired  it  in  both  formats,  with  one  group  preferring  paper  and  one  group 
preferring  digital.  The  most  commonly  desired  features  in  a  digital  training  aid  were  a  glossary, 
information  on  how  to  correct  shooting  errors,  and  to  embed  the  training  aid  in  a  modular 
soldier’s  mobile  application  that  would  address  other  topics  such  as  additional  weapons, 
navigation,  and  communications.  All  these  features  were  identified  by  three  different  groups. 
Finally,  the  four  of  the  groups  were  emphatic  that  a  training  aid  should  not  be  used  to  reduce  their 
range  time  and  three  groups  were  similarly  adamant  that  a  digital  application  is  an  aid  to,  and  not 
a  substitute  for,  an  experienced  and  competent  instructor. 

Three  themes  emerged  from  the  focus  group  discussions.  First,  the  soldiers  were  nearly 
unanimous  in  seeing  a  need  for  a  coaching  application  on  a  mobile  device.  They  generally  saw 
the  mobile  device  as  reference  material  for  those  with  direct  responsibility  for  coaching.  They 
were  adamant  that  the  device  be  simply  an  aide  memoire,  and  that  it  should  not  serve  as  a 
substitute  for  a  fully  qualified  coach.  Second,  they  expected  that  a  mobile  device  version  of  the 
guide  would  be  more  than  a  digital  document.  They  believed  that  multimedia  depictions  of 
correct  behaviour,  incorrect  behaviour,  and  remedial  interventions  would  be  significant 
enhancements  to  the  guide.  In  addition,  they  envisioned  that  such  a  guide  would  be  part  of  a  suite 
of  similar  infantry  reference  material  residing  on  the  mobile  device.  The  third  theme  was 
sensitivity  to  the  strengths  and  weaknesses  of  printed  and  digital  materials.  Digital  devices  were 
seen  as  powerful  but  fragile,  expensive  and  power-limited  whereas  printed  materials  were  seen  as 
durable,  cheap,  and  reliable  with  limited  capabilities  relative  possible  digital  systems.  Most  focus 
groups  did  not  see  these  differences  as  grounds  for  a  choice  between  formats,  but  rather  as 
guidance  for  selecting  from  both  available  formats  according  to  their  immediate  circumstances. 
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4  Conclusion 


The  experiment  examined  the  use  and  effect  of  immediately  available  small  arms  coaching 
training  material  with  the  goal  of  understanding  the  effectiveness  of  this  type  of  training  if  it  were 
delivered  as  part  of  a  training  capability  embedded  in  a  future  rifle.  A  second  goal  was  to 
determine  if  soldiers  would  avail  themselves  of  the  anywhere  -  anytime  training  capability  that 
would  be  provided  by  embedded  training.  A  final  goal  was  to  understand  the  usability  of  the 
systems  provided  examined. 

The  soldiers  did  not  use  the  training  capability  outside  the  experiment.  They  had  the  guide  with 
them  for  five  days  and  four  nights  while  they  were  at  DRDC  Toronto.  Their  scheduled 
participation  in  several  experiments  each  day  provided  them  with  regular  periods  where  they  were 
not  otherwise  engaged.  They  were  away  from  their  home  station  so  they  had  no  obligations  in  the 
evening.  Even  with  the  knowledge  that  they  would  again  be  tested  on  marksmanship  errors  on  the 
Friday  of  the  week,  most  soldiers  looked  at  the  guide  only  once  on  Monday  and  not  again.  This 
finding  does  not  support  the  idea  that  soldiers  will  spontaneously  engage  in  self-study  if  provided 
with  the  materials  and  time.  The  reason  they  did  not  do  so  is  unclear.  It  may  be  they  believed  they 
already  mastered  the  material,  that  the  material  was  not  valuable,  or  that  they  were  not  motivated 
to  learn  the  material.  The  implication  of  this  result  is  that  the  mere  provision  of  training  capability 
in  an  embedded  or  mobile  device  is  not  sufficient  to  produce  learning  and  that  the  early 
expectations  in  this  regard  (BenMoussa,  2003;  Virvou  &  Alepis,  2005)  may  not  be  easily 
realized.  Soldiers  will  not  automatically  use  personal  or  spare  work  time  to  train.  Direction  and 
motivation  are  still  required. 

The  soldiers  did  not  use  the  guide  as  a  job  aid  to  assist  them  during  the  actual  task  when  given  the 
opportunity  to  do  so,  despite  their  imperfect  ability  to  identify  the  shooting  errors. 

The  soldier’s  identification  of  shooting  errors  became  more  accurate  over  the  course  of  the  two 
tests  provided  on  Monday  and  the  two  tests  on  Friday.  Given  that  they  did  not  make  substantial 
use  of  the  guide  between  the  pairs  of  tests,  it  is  apparent  that  learning  occurred  as  a  result  of  the 
testing  itself,  a  phenomenon  called  the  testing  effect  (Roediger  &  Karpicke,  2006).  Consequently, 
the  results  obtained  do  not  provide  evidence  regarding  the  efficacy  of  the  guide  as  presented  in 
digital  and  paper  form.  However,  the  nature  of  the  tests  developed  from  the  guide,  the  videos  of 
errors  with  the  instruction  to  detect  the  error,  is  exactly  the  kind  of  multimedia  training  material 
that  would  be  delivered  in  an  embedded  or  mobile  training  device.  As  such,  this  provides  strong 
evidence  of  the  effectiveness  of  this  approach  to  training  small  arms  coaching. 

Usability  questionnaire  data  revealed  that  the  simple  digital  guide  was  found  to  be  at  least  as 
usable  as  the  printed  version.  The  implication  is  possible  to  provide  embedded  or  mobile  training 
material  to  dismounted  soldiers  satisfactorily. 

Finally,  it  was  found  that  soldiers  were  receptive  to  using  a  digital  training  guide  for  small  arms 
coaching  but  they  are  wary  that  the  introduction  of  a  digital  training  aid  might  be  used  as 
justification  to  reduce  their  time  on  the  range  or  to  provide  unqualified  coaches. 
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4.1  Recommendations 


On  the  basis  of  the  preceding  findings,  the  following  recommendations  are  made. 

4.1.1  Evaluate  a  prototype  mobile  application  for 
marksmanship  coaching 

A  prototype  mobile  application  for  marksmanship  coaching  should  be  developed  for  the  Army. 
Developed  in  consultation  with  Canadian  Army  small  arms  experts  and  trainers,  the  application 
should  be  provided  to  a  group  of  soldiers  slated  to  undertake  small  arms  instruction  training.  The 
experiment  would  record  their  usage  of  the  system  and  their  performance  on  the  small  arms 
instruction  course  for  comparison  to  a  similar  group  not  provided  with  the  mobile  learning 
application.  Ideally  an  additional  group  would  be  provided  with  a  paper  version  of  the  training  aid 
to  determine  if  the  computing  capabilities  of  the  mobile  device  assist  in  training.  The  practical 
benefit  of  this  action  will  be  to  improve  army  marksmanship  coaching.  A  secondary  benefit 
would  be  a  better  understanding  of  the  circumstances  of  when  and  where  soldiers  will  use 
embedded  or  mobile  training  applications.  This  knowledge  would  further  inform  instructional 
design  decisions  by  Training  Development  Officers,  enabling  them  to  develop  more  efficient  training. 

4.1.2  Field  embedded  or  mobile  training  systems  with 
performance  support 

To  ensure  independent  use  of  embedded  or  mobile  training  systems,  many  soldiers  may  require 
support  in  the  form  of  direction  of  when  and  why  to  train  as  well  as  motivation  to  do  so.  If  this  is 
not  done  in  conjunction  with  the  issuing  of  the  training  capability,  such  as  being  loaded  on  a 
formal  course,  it  may  be  beneficial  to  provide  informal  structure  and  motivation.  Recent  research 
describes  how  this  might  be  accomplished  using  online  discussion  groups  or  peer  mentoring 
(Boyle,  Kwon,  Ross,  &  Simpson,  2010). 
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Annex  A  Shooting  error  checklist 


Error 

1  2 

3 

4  S 

6 

7  8 

Firing  Hand  /  Arm 

Canted  rifle 

Butt  not  tight  against  shoulder 

Firing  elbow  horizontal  ("chicken  wing") 

Firing  hand  too  loose 

Firing  hand  too  tight 

Firing  hand  too  low  on  pistol  grip 

Fingertip  on  trigger 

Slapping  and  releasing  trigger 

Trigger  finger  canted  up  or  down 

Trigger  finger  position  causes  rotated  wrist 

Non  -  Firing  Hand  /  Arm 

Non-firing  hand  too  tight 

Rifle  passively  resting  on  non-firing  hand 

Rifle  not  resting  at  V  of  thumb  and  palm 

Bone-to-bone  or  flesh-to-flesh  on  knee 

Excess  wobble  of  muzzle 

Supporting  &  Control  Arm 

Elbows  too  close  together 

Elbows  too  far  apart 

Head 

Raising  head  between  shots 

Raised  head  to  look  at  target 

Inconsistent  head  position 

Not  looking  through  center  of  firing  eye 

Flinching 

Closed  eyes  during  firing 

Legs 

Legs  not  at  90‘  (kneeling) 

Legs  bowed  (prone) 

Non-Firing  leg  cocked  (prone) 

On  toes  (prone) 

Torso 

Breathing  while  firing 

Over  -  breathing 

Natural  point  of  aim  not  aligned  with  target 

Not  directly  behind  rifle 

Not  leaning  forward 

Buttocks  not  resting  on  firing  side  foot  as  much  as  possible 

Equipment  improperly  worn 

Armour  /  helmet  interference 

Whole  Body 

Fidgeting 

No  error 

Figure  A.l:  The  shooting  error  checklist. 
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Annex  B  Usability  instruments 


Table  B.l:  The  abbreviated  version  of  the  mobile  phone  usability  questionnaire 

(Ryu  7  Smith -Jacks on,  2005). 

Mobile  Phone  Usability  Questionnaire  (Abbreviated) 


Please  circle  the  best  answer. 

Galaxy  Tab  Apple  iPad 

Circle  the  device  you  used. 


Apple  iPad  Mini  Samsung  Large 
Tablet 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  it  easy  to  learn  to  operate  this  product? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  using  this  product  sufficiently  easy? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Have  the  user  needs  regarding  this  product  been  sufficiently  taken  into  consideration? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  it  relatively  easy  to  move  from  one  part  of  a  task  to  another? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Can  all  operations  be  carried  out  in  a  systematically  similar  way? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Are  the  operations  of  this  product  simple  and  uncomplicated? 
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Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Does  this  product  enable  the  quick,  effective,  and  economica 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

performance  of  tasks? 


Is  it  easy  to  access  the  information  that  you  need  from  the  product? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  the  organization  of  information  on  the  product  screen  clear? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Does  product  have  all  the  functions  and  capabilities  you  expect  it  to  have? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Are  the  color  coding  and  data  display  compatible  with  familiar  conventions? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  it  easy  for  you  to  remember  how  to  perform  tasks  with  this  product? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2  3  4  5  6 

7 

Is  the  inter 

'ace  with  this  product  clear  and  understandable? 

Strongly 

Disagree 

Strongly 

Agree 

1 

2  3  4  5  6 

7 

Are  the  characters  on  the  screen  easy  to  read? 
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Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Does  interacting  with  this  product  require  a  lot  of  mental  effort? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Can  you  regulate,  control,  and  operate  the  product  easily? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  it  easy  to  navigate  between  hierarchical  menus,  pages,  and  screen? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  the  backlighting  feature  for  the  keyboard  and  screen  helpful? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Are  the  command  names  meaningful? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  discovering  new  features  sufficiently  easy? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  the  HELP  information  given  by  this  product  useful? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  the  presentation  of  system  information  sufficiently  clear  and  understandable? 
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Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  it  easy  to  take  corrective  actions  once  an  error  has  been  recognized? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  feedback  on  the  completion  of  tasks  clear? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Does  the  product  give  all  the  necessary  information  for  you  to  use  it  in  a  proper  manner? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2  3  4  5  6 

7 

Is  the  bolding  of  commands  or  other  signals  helpfu 

? 

Strongly 

Disagree 

Strongly 

Agree 

1 

2  3  4  5  6 

7 

Does  the  HELP  function  define  aspects  of  the  product  adequately? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  this  product's  size  convenient  for  transportation  and  storage? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  using  this  product  frustrating? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  this  product  attractive  and  pleasing? 
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Strongly 

Disagree 

Strongly 

Agree 

1 

2  3  4  5  6 

7 

Do  you  fee 

comfortable  and  confident  using  this  product? 

Strongly 

Disagree 

Strongly 

Agree 

1 

2  3  4  5  6 

7 

Does  the  color  of  the  product  make  it  attractive? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Does  the  brightness  of  the  product  make  it  attractive? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Are  pictures  on  the  screen  of  satisfactory  quality  and  size? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  the  number  of  colors  available  adequate? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Are  the  components  of  the  product  are  well-matched  or  harmonious? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Do  you  fee 

excited  when  using  this  product? 

Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Would  you  miss  this  product  if  you  no  longer  had  it? 
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Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Are  you/would  you  be  proud  of  this  product? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Does  carrying  this  product  make  you  feel  stylish? 

Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  the  organization  of  the  menus  sufficiently  logica 

? 

Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  the  design  of  the  graphic  symbols,  icons  and  labels  on  the  icons  sufficiently  relevant? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Does  the  product  provide  index  of  data? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  the  highlighting  on  the  screen  helpful? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Are  the  HOME  and  MENU  buttons  sufficiently  easy  to  locate  for  all  operations? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Are  the  response  time  and  information  display  fast  enough? 
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Strongly 

Disagree 


Strongly 
Agree 


1 

2 

3 

4 

5 

6 

7 

Has  the  product  at  some  time  stopped  unexpectec 

ly? 

Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  the  amount  of  information  displayed  on  the  screen  adequate? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  the  way  product  works  overall  consistent? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  the  data  display  sufficiently  consistent? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Does  the  product  support  the  operation  of  all  the  tasks  in  a  way  that  you  find  useful? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  the  product  reliable,  dependable,  and  trustworthy? 


Strongly 

Disagree 

Strongly 

Agree 

1 

2 

3 

4 

5 

6 

7 

Is  it  sufficiently  easy  to  operate  keys  with  one  hand? 
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Table  B.  2:  The  system  usability  scale  (Brooke,  1996). 

System  Usability  Scale 

Participant  # _ 

Please  respond  to  the  following  statements  about  the  Marksmanship  Guide  by  marking  the 
box  that  most  closely  matches  your  opinion  (1  =  strongly  disagree;  3  =  neither  agree  nor 
disagree;  5  -  strongly  agree) _ _ 


1 .  1  think  that  1  would  like  to  use  this 
system  frequently 

Strongly  Strongly 

Disagree  Agree 

□  □  □  □  □ 

1  2  3  4  5 

2.  1  found  the  system  unnecessarily 
complex 

Strongly  Strongly 

Disagree  Agree 

□  □  □  □  □ 

1  2  3  4  5 

3.  1  thought  the  system  was  easy  to  use 

Strongly  Strongly 

Disagree  Agree 

□  □  □  □  □ 

1  2  3  4  5 

4.  1  think  that  1  would  need  the  support  of 
a  technical  person  to  be  able  to  use  this 
system 

Strongly  Strongly 

Disagree  Agree 

□  □  □  □  □ 

1  2  3  4  5 

5.  1  found  the  various  functions  in  this 
system  were  well  integrated 

Strongly  Strongly 

Disagree  Agree 

□  □  □  □  □ 

1  2  3  4  5 

6.  1  thought  there  was  too  much 
inconsistency  in  this  system 

Strongly  Strongly 

Disagree  Agree 

□  □  □  □  □ 

1  2  3  4  5 

7.  1  would  imagine  that  most  people 
would  learn  to  use  this  system  very 
quickly 

Strongly  Strongly 

Disagree  Agree 

□  □  □  □  □ 

1  2  3  4  5 

8.  1  found  the  system  very  cumbersome 
to  use 

Strongly  Strongly 

Disagree  Agree 

□  □  □  □  □ 

1  2  3  4  5 

9.  1  felt  very  confident  using  the  system 

Strongly  Strongly 

Disagree  Agree 

□  □  □  □  □ 

1  2  3  4  5 

10.  1  needed  to  learn  a  lot  of  things 
before  1  could  get  going  with  this  system 

Strongly  Strongly 

Disagree  Agree 

□  □  □  □  □ 

1  2  3  4  5 
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Table  B.3:  The  context  and feature  questionnaire. 

Context  and  Feature  Questionnaire 


Participant  # _ 

Please  describe  the  two  most  useful  contexts  for  using  a  publication  for  marksmanship. 


Context  Example:  At  the  beach 

Context  1: 

Context  2: 

Goal 

Goal 

Goal 

3  Study  by  an  small  arms  user 

□  Study  by  an  small  arms  user 

□  Study  by  an  small  arms  user 

□  Study  by  a  small  arms  instructor 

□  Study  by  a  small  arms  instructor 

□  Study  by  a  small  arms  instructor 

□  While  coaching  a  buddy 

□  While  coaching  a  buddy 

□  While  coaching  a  buddy 

□  While  providing  small  arms  instruction 

□  While  providing  small  arms  instruction 

□  While  providing  small  arms  instruction 

Other: 

Other 

Other: 

Location 

Location 

Location 

□  Indoors 

□  Indoors 

□  Indoors 

3  Outdoors 

□  Outdoors 

□  Outdoors 

□  Vehicle 

□  Vehicle 

□  Vehicle 

Type 

Type 

Type 

□  Residential 

□  Residential 

□  Residential 

□  Work  /School 

□  Work /School 

□  Work  /  School 

3  Recreational  /  Entertainment 

□  Recreational  /  Entertainment 

□  Recreational  /  Entertainment 

Other: 

Other. 

Other: 

Environment 

Environment 

Environment 

3  Very  bright 

□  Very  bright 

□  Very  bright 

□  Dark 

□  Dark 

□  Dark 

3  Wet 

□  Wet 

□  Wet 

3  Dirty  /  Dusty 

□  Dirty  /  Dusty 

□  Dirty  /  Dusty 

□  Hot 

□  Hot 

□  Hot 

□  Cold 

□  Cold 

□  Cold 

3  Comfortable  temperature 

□  Comfortable  temperature 

□  Comfortable  temperature 

3  Noisy 

□  Noisy 

□  Noisy 

□  Very  Quiet 

□  Very  Quiet 

□  Very  Quiet 

Other: 

Other 

Other: 

Movement 

Movement 

Movement 

3  Stationary 

□  Stationary 

□  Stationary 

□  Walking 

□  Walking 

□  Walking 

□  Moving  in  vehicle 

□  Moving  in  vehicle 

□  Moving  in  vehicle 

Other  tasks 

Other  tasks 

Other  tasks 

□  Simultaneous 

□  Simultaneous 

□  Simultaneous 

□  Switching 

□  Switching 

□  Switching 

3  No  other  tasks 

□  No  other  tasks 

□  No  other  tasks 

Other  task: 

Other  task: 

Other  task: 

Duration  per  use 

Duration  per  use 

Duration  per  use 

□  Less  than  5  min 

□  Less  than  5  min 

□  Less  than  5  min 

3  5-9  min 

□  5-9  min 

□  5-9  min 

□  10-14  min 

□  10-14  min 

□  10-14  min 

□  15  -19  min 

□  15  -19  min 

□  15  -19  min 

□  20  min  or  more 

□  20  min  or  more 

□  20  min  or  more 

Time  when  used 

Time  when  used 

Time  when  used 

□  Before  work 

□  Before  work 

□  Before  work 

□  At  work 

□  At  work 

□  At  work 

□  During  breaks 

□  During  breaks 

□  During  breaks 

3  After  work 

□  After  work 

□  After  work 

Sequence 

Sequence 

Sequence 

□  Before  other  task 

□  Before  other  task 

□  Before  other  task 

□  During  other  task 

□  During  other  task 

□  During  other  task 

□  After  other  task 

□  After  other  task 

□  After  other  task 

3  Not  sequenced 

□  Not  sequenced 

□  Not  sequenced 

Other  task: 

Other  task: 

Other  task: 

Social  Situation 

Social  Situation 

Social  Situation 

□  Nobody  else  present 

□  Nobody  else  present 

□  Nobody  else  present 

□  Strangers  present 

□  Strangers  present 

□  Strangers  present 

3  Amongst  family  /  friends 

□  Amongst  family  /  friends 

□  Amongst  family  /  friends 

□  Amongst  co-workers 

□  Amongst  co-workers 

□  Amongst  co-workers 

□  Interactively  with  others 

□  Interactively  with  others 

□  Interactively  with  others 

Infrastructure 

Infrastructure 

Infrastructure 

□  WiFi  available 

□  WiFi  available 

□  WiFi  available 

□  Wide  -area  wireless  network 

□  Wide  -area  wireless  network 

□  Wide  -area  wireless  network 

3  Personal  device 

□  Personal  device 

□  Personal  device 

□  Device  provided  through  work 

□  Device  provided  through  work 

□  Device  provided  through  work 
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Features  and  Suggestions 


Please  write  down  your  thoughts  on  what  would  make  a  marksmanship  mobile  app  or  booklet  easy  to  use  and  effective. 
Changes  to  the  system  you  have  used  and  suggestions  for  new  features  are  welcome. 

Some  possible  topics: 


•  Booklet  or  mobile  device  physical  characteristics  (size,  finish,  etc) 

•  Learning  Content  (e.g.  less  material,  more  theory,  less  theory,  longer,  shorter) 

•  Graphics  (videos,  more  pictures,  etc) 

•  Organization  of  the  content 

•  Who  should  use  it,  and  when,  and  where. 

•  Differences  of  where  a  booklet  or  an  app  would  be  better 
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Annex  C  Focus  group  data  summary 


Table  C.l:  Focus  group  response  frequencies. 


Topic 

Responses 

Frequency 

Is  there  a  need  for  a  small  arms  coaching 
training  aid? 

Yes 

6 

No 

3 

Information  is  already  available 
elsewhere 

3 

Are  there  different  individuals  who  need 
the  training  aid? 

There  should  be  a  single  version  with 
different  sections  for  novice  and 
advanced  users 

6 

Does  the  need  for  such  a  training  aid 
differ  between  instructors  or  is  it  for  all 
soldiers? 

There  should  be  a  single  version  with 
different  sections  for  novice  and 
advanced  users 

3 

Does  the  need  for  such  a  training  aid 
differ  between  combat  arms  and  those 
not  in  the  combat  arms? 

Same  version  for  all 

2 

There  should  be  a  single  version  with 
different  sections  for  novice  and 
advanced  users 

3 

Does  the  need  for  such  a  training  aid 
differ  between  the  regular  forces  and  the 
reserves? 

Same  version  for  all 

2 

Do  the  air  force  or  navy  need  for  such  a 
training  aid 

Yes 

5 

At  what  stage  in  a  soldier’s  career  is  a 
small  arms  coaching  training  aid  needed? 

Right  from  the  start 

3 

Only  needed  when  they  advance  in 
rank 

1 
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Topic 

Responses 

Frequency 

Should  the  training  aid  be  provided  in  a 
paper  or  digital  format? 

Paper 

5 

Digital 

5 

Paper  is  more  durable  (dirt,  weather) 

2 

Digital  devices  are  fragile 

2 

Paper  does  not  need  power 

2 

Digital  needs  power 

2 

Paper  documents  are  easy  to  issue  to 
soldiers 

2 

Digital  devices  are  difficult  to  issue  to 
soldiers 

2 

When  is  one  format  better  than  another? 

Digital  content  is  easy  to  obtain 

2 

Paper  content  is  difficult  to  obtain 

1 

Paper  documents  bring  weight  and 
bulk 

2 

Digital  for  clean  conditions,  e.g., 
classroom 

2 

Paper  for  rugged  conditions,  e.g.,  field 

2 

Digital  may  have  less  appeal  to  older 
soldiers 

1 

Digital  may  have  more  appeal  to 
younger  soldiers 

2 

What  features  are  needed  in  a  digital 

Glossary 

3 
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Topic 

Responses 

Frequency 

training  aid? 

Make  the  coaching  training  application 
a  module  in  a  larger  soldier  application 
that  would  include  information  on 
other  weapons,  map  reading, 

navigation,  etc. 

3 

Show  how  to  correct  shooting  errors 

3 

Provide  3D  /  360  degree  view  of  firers 

2 

Use  moving  images  to  depict  correct 
and  incorrect  shooting 

2 

Provide  an  efficient  search  function 

2 

Enable  use  of  the  device  camera  for 
capturing  and  debriefing  performance 
of  the  firer 

2 

Add  close  quarter  battle  techniques 

1 

Address  physiology  /  stress  of  shooting 

1 

Adjusting  sights 

1 

Availability  on  DWAN 

1 

Ballistics  data  for  space  beyond  the 
target 

1 

Ballistics  data  for  target  effects 
/penetration 

1 

Care  of  weapons  in  weather 

1 

Different  sections  for  trainer  /  trainee 

1 
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Topic 

Responses 

Frequency 

Discuss  acceptable  personal  variations 
in  shooting  technique 

1 

Entertaining  features  and  soldier- 
friendly  language 

1 

Explain  grouping  and  zeroing 

1 

Label  images  with  information 
indicating  correct  or  incorrect 
technique 

1 

Left  handed  firer  coaching  information 

1 

Range  estimation  techniques 

1 

Shot  analysis  information 

1 

Specifics  of  combat  shooting  as 
compared  to  range  shooting 

1 

Video  of  how  rifle  works 

1 

Wind  /  weather  effects  on  shooting 

1 

What  should  not  be  included? 

CQB  should  be  treated  separately 

2 

Do  not  provide  excess  information 

3 

How  should  the  training  aid  you  used  be 
modified? 

Canadianize  the  content 

3 

Observe  digital  device  conventions 
like  swiping  to  turn  pages 

2 

Are  there  other  things  to  offer  or  other 

No  excuse  to  take  away  range  time 

4 
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Topic 

Responses 

Frequency 

considerations? 

Should  not  be  a  crutch  or  substitute  for 
competent  and  qualified  instructors 

3 

Make  available  for  use  in  downtime 

1 
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Annex  D  Focus  group  protocol 


Table  D.l:  Focus  group  protocol. 

1  Introduction 

Thank  you  for  completing  the  questionnaires.  For  the  final  part  of  this 
experiment,  I  would  like  to  lead  a  discussion  on  your  thoughts  about  using 
a  mobile  app  or  booklet  to  help  with  marksmanship  knowledge.  This  is  a 
chance  for  us  to  discuss  some  of  the  ideas  you  have  come  up  with  on  your 
own. 

2  Ground  Rules 

Before  we  begin,  I  would  like  to  review  a  few  ground  rules  for  the 
discussion. 

(2  minutes) 

•  1  am  going  to  ask  you  several  questions;  we  do  not  have  to  go  in 
any  particular  order  but  we  do  want  everyone  to  take  part  in  the 
discussion.  We  ask  that  only  one  person  speak  at  a  time. 

•  Feel  free  to  treat  this  as  a  discussion  and  respond  to  what  others 
are  saying,  whether  you  agree  or  disagree.  We're  interested  in 
your  opinions  and  whatever  you  have  to  say  is  fine  with  us.  We 
are  just  asking  for  your  opinions  based  on  your  own  personal 
experience.  We  are  here  to  learn  from  you. 

•  Don't  worry  about  having  a  different  opinion  than  someone  else. 
But  please  do  respect  each  other's  answers  or  opinions. 

•  We  are  taking  notes  because  we  don't  want  to  miss  any  of  your 
comments. 

•  Does  anyone  have  any  questions  before  we  start? 

3  Group  Discussion 
-  Who  Needs  This? 

The  first  question  is  whether  there  is  a  need  for  an  app  or  a  booklet  to  help 
people  identify  shooting  errors. 

•  Is  there  a  need  for  it? 

(10  minutes) 

•  Are  there  different  people  with  different  needs? 

•  Individual  soldiers? 

•  Small  Arms  Instructors? 

•  Combat  vs  Non-Combat  Arms? 

•  Air  Force  and  Navy? 

•  Regular  vs  Reserves? 

•  At  what  stage  in  their  military  career? 

4  Group  Discussion 
-  What  format? 

Our  next  topic  is  the  format  of  the  material.  This  could  be  produced  as  a 
booklet  or  an  app.  The  booklet  is  always  usable  and  is  does  not  need  power 
or  a  device,  but  takes  space  and  weight.  An  app  can  provide  multimedia 

(5  minutes) 

and  other  features,  and  has  no  space  or  weight  if  you  already  have  a 
device. 

•  What  do  you  think  is  better? 

•  When  is  one  better  than  another? 

5  Group  Discussion 
-  What  features? 

Our  final  topic  is  to  think  about  what  a  version  for  the  CAF  should  be  like. 

•  If  you  were  building  it,  what  would  you  want  in  it? 
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(15  minutes) 

•  What  should  it  not  include? 

•  How  should  the  one  you  tried  be  modified? 

6  Final  Thoughts 

That  is  the  end  of  my  questions. 

•  Are  there  any  other  things  you  would  like  to  offer  or  other  things 
that  should  be  considered? 

7  Wrap  up 

Thanks  for  participating  in  the  discussion.  That  brings  us  to  the  end  of  the 
experiment.  [DEBRIEF  AND  DISMISS1 
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List  of  symbols/abbreviations/acronyms/initialisms 


CAF 

Canadian  Armed  Forces 

CAN-LEAP 

Canadian  Load  Effects  Assessment  Program 

CER 

Combat  Engineer  Regiment 

CQB 

Close  Quarter  Battle 

DR  DC 

Defence  Research  and  Development  Canada 

DWAN 

Defence  Wilde  Area  Network 

ET 

Embedded  Training 

FSAR 

Future  Small  Arms  Research 

PWT 

Personal  Weapons  Test 

RCR 

Royal  Canadian  Regiment 
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